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RETRIEVAL  OF  KNOWLEDGE  THROUGH  ALGORITHMIC  DECOMPOSITION 


Decision  makers  often  need  to  know  the  value  of  a  particular 
quantity,  such  as  "How  much  money  did  our  family  spend  on  food  last 
year?"  or  "What  is  the  Soviet  troop  strength  in  Cuba?"  Sometimes  the 
quantitative  value  is  readily  obtainable  from  a  computer  database, 
encyclopedia,  reference  source,  or  expert.  In  other  cases,  one  may  have 
read  or  been  told  the  value  so  that  it  can  be  accurately  and  confidently 
retrieved  from  memory  when  it  is  needed. 

There  are  some  quantities,  however,  that  may  not  exist  in  any 
reference  sources  or  may  be  available  only  in  sources  that  are  difficult 
to  locate,  prohibitively  costly  to  obtain,  take  too  long  to  use,  or 
contain  only  vague  and  partial  knowledge.  Faced  with  this  situation, 
the  best  one  can  do  is  make  an  estimate  of  the  needed  quantity  based  on 
the  informaMon  resources  at  one’s  disposal.  In  principle,  those 
resources  could  extend  to  decision  support  systems,  computer  databases, 
information  libraries,  and  the  judgments  of  expertB.  In  practice, 
however,  estimates  are  often  based  on  whatever  relevant  knowledge  an 
estimator  is  able  to  obtain. 

Even  when  a  computer  database  is  available,  its  contents  may 
contain  information  related  to  the  quantity  called  for,  but  not  the 
quantity  itself.  Indeed,  making  full  use  of  information  systems  entails 
understanding  both  the  capacity  of  the  system  to  provide  direct  answers 
to  questions  and  its  capacity  to  provide  relevant  information  for 
questions  it  cannot  answer  directly.  Effective  querying  of  the  system 
in  the  latter  case  requires  a  careful  structuring  of  the  user's 
Information  requirements,  the  absence  of  which  can  lead  to  inefficient 
use  of  computer  resources,  Incomplete  information  retrieval,  or 
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erroneous  results.  How  well  one  Is  able  to  exploit  computerised 
information  resources  in  such  situations  is  in  part  dependent  upon  one's 
ability  to  structure  whatever  bite  of  information  the  system  can  provide 
in  a  form  that  is  meaningful  for  the  task,  of  estimating  the  quantity  in 
question.  While  the  substantive  contents  of  that  structuring  are 
potentially  available  from  an  information  system,  the  form  of  the 
structuring  is  usually  left  to  the  idiosyncracies  of  the  individual 
user.  Studying  the  behavioral  properties  of  various  methods  of 
structuring  information  helps  shed  light  on  the  relative  performance 
that  might  be  expected  from  individuals  when  they  are  provided  specific 
guidance  on  how  to  approach  organizing  quantitative  estimation  problems. 

Approaches  to  structuring  knowledge  retrieval  for  the  sake  of 
estimating  an  uncertain  quantity  can  vary  in  form  and  elaboration.  The 
simplest  approach  is  to  consider  what  one  knows  about  a  quantity  ot 
interest  and  intuitively  divine  an  estimate  that  seems  reasonable  in 
light  of  whatever  knowledge  comes  to  mind.  This  wholistic  approach  to 
estimation  relies  heavily  on  the  power  of  unaided  human  cognition.  To 
it6  advantage,  it  is  inexpensive,  portable,  and  represents  the  way  in 
which  people  routinely  deal  with  estimation  problems,  thus  achieving  a 
high  degree  of  psychological  compatibility.  To  its  disadvantage,  memory 
could  prove  to  be  too  impoverished  a  resource  on  which  to  base  an 
estimate  and  may  provide  no  Indication  of  which  information  might  prove 
useful.  Even  when  seemingly  useful  knowledge  is  retrieved,  there  mey  be 
no  way  of  knowing  how  to  combine  disparate  pieces  of  knowledge  into  m 
global  estimate.  Furthermore,  research  on  the  psychology  of  huuan 
judgment  has  repeatadly  demonstrated  that  almpllfying  cognitive 
•tratagiea  can  laad  to  ayatamatic  judgmantal  blaata. 
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An  alternative  to  the  wholistic,  intuitive  approach  is  analysis  or 
decomposition.  This  involves  breaking  up  or  decomposing  a  problem  into 
a  series  of  sub-problems  or  components,  each  of  which  can  be  understood 
more  easily  and  operated  on  separately.  The  components  are  then 
assembled  according  to  a  prescribed  set  of  combination  rules  to  yield  a 
solution,  estimate,  or  prediction.  Decomposition  is  a  divide-and- 
conquer  approach  that  assumes  the  components  of  a  problem  to  be  more 
understandable  and  tractable  than  the  undecomposed  problem. 

A  variant  of  the  decompositlonal  approach  is  the  algorithm.  An 
algorithm  is  a  series  of  steps  or  operations  that,  when  sequentially 
applied,  produce  a  solution  to  a  problem.  Essentially,  algorithms  work 
by  providing  an  unambiguous  procedure  for  solving  problems.  They  help 
structure  what  is  known  about  a  problem,  point  out  what  is  not  known, 
and  specify  the  rules  by  which  information  should  be  combined. 

The  research  summarized  here  was  designed  to  explore  the 
effectiveness  of  algorithms  as  an  aid  for  estimating  unknown  quantities. 
In  order  to  evaluate  effectiveness,  we  have  used,  in  our  research, 
quantities  for  which  we  have  accurate  estimates,  gained  from  almanacs 
and  other  such  sources.  However,  we  have  used  quantities  for  which 
people  are  unlikely  to  have  ready  estimates,  such  as  the  number  of 
pounds  of  potato  chips  consumed  annually  in  the  United  States. 

In  our  first  study  (MacGregor,  Lichtenstein  &  Slovic,  in  press),  we 
selected  sixteen  questions  about  uncertain  quantities  like  MHow  many 
people  are  employed  by  hospitals  in  the  U.S.?"  These  sixteen  questions 
were  presented  under  five  different  aiding  conditions.  The  most-aided 
group  of  subjects  were  given  complete  algorithms,  created  by  the 
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experimenters,  for  each  question  (for  example,  Average  number  of 
hospitals  per  state  times  number  of  states  times  average  number  of 
employees  per  hospital  equals  number  of  people  employed  by  hospitals). 
These  subjects  were  asked  to  make  estimates  for  all  the  parts  of  the 
algorithm  and  to  combine  the  parts  as  indicated  to  arrive  at  an  estimate 
of  the  desired  quantity.  The  second  group  of  subjects  were  given  the 
same  algorithm  without  any  Indication  of  how  to  combine  the  parts.  They 
were  asked  to  make  an  estimate  for  each  component  and  then  estimate  the 
desired  quantity.  The  third  group  of  subjects  were  asked  to  list 
components  or  factors  they  thought  were  relevant  to  the  question,  to 
make  an  estimate  of  each  item  on  their  list,  and  finally,  to  estimate 
the  desired  quantity.  The  fourth  group  of  subjects  were  asked  to  make 
the  same  sort  of  list,  but  they  were  not  asked  to  make  estimates  of  each 
item  before  making  an  estimate  of  the  desired  quantity.  The  fifth  group 
received  no  aid. 

The  results  of  this  experiment  showed  that  with  increasing 
structure  of  the  aid,  the  subjects  showed  improved  performance  in  terms 
of  both  accuracy  and  consistency  across  subjects. 

For  factual  questions  like  the  ones  used  in  this  experiment,  a 
decision  maker  would  not  normally  have  an  algorithm  readily  available. 

In  our  second  experiment  (Lichtenstein  &  MacGregor,  1984),  we  explored 
the  effectiveness  of  algorithmic  aids  on  a  class  of  numerical  problems 
for  which  an  algorithm  is  known.  These  problems  are  problems  involving 
the  combining  of  two  pieces  of  probabilistic  information.  They  ara 
presented  as  story  problems,  like  the  following: 
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A  light  bulb  factory  uses  a  scanning  device  which  is 
supposed  to  put  a  mark  on  each  defective  bulb  it  spots  in  the 
aasembl>  line.  Eighty-five  percent  (85X)  of  the  light  bulbs 
on  the  line  are  OK;  the  remaining  151  are  defective. 

The  scanning  device  is  known  to  be  accurate  in  80Z  of  the 
decisions,  regardless  of  whether  the  bulb  is  acually  OK  or 
actually  defective.  That  is,  when  a  bulb  is  good,  the  scanner 
correctly  identifies  it  as  good  801  of  the  time.  When  a  bulb 
is  defective,  the  scanner  correctly  marks  it  as  defective  80Z 
of  the  time. 

Suppose  someone  selects  one  of  the  light  bulbs  from  the 
line  at  random  and  gives  it  to  the  scanner.  The  scanner  marks 
this  bulb  as  defective. 

What  is  the  probability  that  this  bulb  is  really 
defective? 

We  prepared  two  such  problems,  the  light  bulb  problem  shown  above 
and  a  structurally  Identical  problem  involving  a  screening  test  for  the 
reading  disability,  dyslexia,  which  has  a  false  alarm  rate  and  a  false 
negative  rate  both  equal  to  5X.  One  of  these  problems  was  presented  to 
three  groups  of  subjects.  One  group  received  no  aid.  A  second  group 
were  given  an  algorithm  to  solve  the  problem  but  no  explanation  as  to 
why  the  algorithm  represented  the  correct  solution.  The  third  group 
received  a  lengthy  tutorial  explaining  how  to  solve  such  problems. 

After  several  intervening,  unrelated  tasks,  all  subjects  were  given  the 
second  problem,  without  any  indication  that  it  was  related  to  the 
previous  task  and  without  any  aid. 
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The  results  showed  that  the  group  given  the  unexplained  algorithm 
showed  better  performance  than  did  the  control  group  on  the  lirst 
problem  but  that  this  improvement  did  not  extend  to  the  second  problem. 
In  contrast,  the  subjects  given  the  lengthy  tutorial  showed  improvement 
for  both  the  first  and  second  problems.  These  results  suggest  that 
algorithms  used  without  understanding  are  of  limited  help. 

Most  of  the  uncertain  quantities  that  decision  makers  face  arise 
without  the  availability  of  an  algorithm  provided  by  someone  else. 
Usually,  decision  makers  must  create  their  own  algorithms.  The 
effectiveness  of  algorithms  in  such  situations  was  explored  in  two 
reports.  For  both  studies,  the  subjects  were  given  questions  about 
uncertain  quantities  and  answers  to  those  questions.  The  subjects  were 
assured  that  the  answer  provided  by  the  experimenters  was  a  wrong 
answer;  the  subjects'  task  was  to  decide  whether  the  given  answer  was 
too  high  or  too  low. 

In  the  first  of  these  two  studies  (Lichtenstein,  MacGregor  & 

Slovic,  1967)  wo  compared  two  types  of  instructions  in  how  to  create 
algorithms.  In  one  case  we  told  the  subjects  to  momentarily  disregard 
our  answer  and  to  build,  from  facts  they  knew  or  could  estimate,  an 
algorithm  that  would  produce  an  estimate.  Then  they  should  compare 
their  estimate  with  our  answer  to  decide  whether  our  answer  was  too  high 
or  too  low.  In  the  other  case  we  told  subjects  to  start  with  our  answer 
and  decompose  it,  using  facts  they  knew  or  could  estimate,  to  arrive  at 
an  implication  of  our  answer  that  they  could  directly  judge,  to  decide 
whether  our  answer  was  too  high  or  two  low. 


The  resales  of  this  study  showed  that,  although  subjects  could 


create  alborithms  most  of  the  time,  there  was  only  modest  improvement  in 
accuracy  due  to  creating  algorithms*  There  was  no  difference  in 
accuracy  as  a  function  of  che  two  types  of  instruction,  although 
subjects  more  often  used  the  former  method  than  the  latter.  Further 
analyses  showed  that  subjects  were  often  misled  by  their  own 
misinformation,  such  as  believing  that  the  population  of  the  U.S. 
exceeds  three  billion  people. 

Ti.e  second  study  about  creating  algorithms  (Lichtenstein  & 

Weathers,  1987)  explored  the  effectiveness  of  asking  people  to  create  an 
algorithm  and  to  use  it  twice,  once  to  arrive  at  a  reasonably  low 
estimate  of  the  desired  quantity  and  again  to  arrive  at  a  reasonably 
high  estimate.  The  results  of  this  study  showed  no  significant 
improvement  in  accuracy  due  to  this  method.  WTien  subjects  gave  two 
estimates,  both  of  which  were  larger  than  the  given  answer,  they 
naturally  concluded  that  our  answer  was  too  low.  This  led  to  excellent 
accuracy,  of  course,  when  our  answer  was  indeed  too  low.  But  the 
subjects  sometimes  gave  two  estimates  that  were  both  larger  than  the 
given  answer  even  when  our  answer  was  in  fact  too  high.  This  situation 
happened  often  enough  that  the  overall  performance  was  no  better  than 
performance  without  algorithms. 

Two  further  reports  looked  in  depth  at  the  difficulties  subjects 
have  in  creating  algorithms.  The  first  (Lichtenstein  &  MacGregor,  1987) 
gave  a  detailed  analysis  of  the  types  of  arithmetic  errors  made  by  our 
subjects.  Only  22*.  of  our  subjects  were  able  to  complete,  without 
arithmetic  or  copying  errors,  a  straightforward  13-step  algorithm  (one 


that  required  no  estimation  on  the  part  of  subjects).  Although  thio 
result  cannot  be  generalized  to  the  population  as  a  whole ,  it  does 
suggest  that  builders  of  decision  aids  exercise  great  caution  in 
assuming  that  their  users  have  even  the  most  basic,  elementary 
arithmetic  skills. 

A  similar  cautionary  tale  can  be  drawn  from  the  next  report 
(Lichtenstein ,  1987),  which  explored  subjects'  knowledge  of  ordinary 
facts.  For  example,  only  about  40%  of  the  subjects,  students  at  a  state 
university,  estimated  the  population  of  the  U.  S.  as  being  between  200 
million  and  300  million.  Only  33%  could  correctly  state  how  many  feet 
there  are  in  a  mile,  alt  uugh  76"  knew  how  many  cups  are  in  a  quart. 

Taken  together,  these  results  suggest  substantial  problems  in 
designing  decision  aids  based  on  algorithmic  decomposition,  not  because 
the  principles  of  creating  algorithms  are  hard  to  learn  but  because  the 
ast-rs  ui;  1m  to.,  jften  mislead  by  their  own  misinformation  and  their 
la--,  of  arithmetic  skills.  This  leads  us  to  believe  that  the  use  or 
such  decision  aids  might  best  be  limited  to  situations  in  which  experts 
can  carefully  and  de 1 i be r a t e ly  design  algorithms  for  use  in  situations 
in  which  computational  aids  are  also  available.  In  our  final  paper  for 
this  project  (Lichtenstein  &  Slovic,  1987),  therefore,  we  discussed  two 
possible  approaches  to  the  design  of  an  expert  system  using  algorithmic 
decomposition.  To  contrast  the  two  epproaches,  one  arising  from  the 
field  of  artificial  intelligence,  the  other,  from  decision  analysis,  we 
took  as  our  example  the  design  of  an  expert  system  to  predict 
dangerousness  . 
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